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Abstract 

We consider a Wright-Fisher diffusion {x{t)) whose current state cannot be observed 
directly. Instead, at times ti < t2 < . . ., the observations y{ti) are such that, given 
the process {x{t)), the random variables {yiti)) are independent and the conditional 
distribution of y{ti) only depends on x{ti). When this conditional distribution has 
a specific form, we prove that the model {{x(ti),y{ti)),i > 1) is a computable filter 
in the sense that all distributions involved in filtering, prediction and smoothing are 
exactly computable. These distributions are expressed as finite mixtures of parametric 
distributions. Thus, the number of statistics to compute at each iteration is finite, but 
this number may vary along iterations. 
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1 Introduction 



Consider a large population composed of two types of individuals A and a. Suppose that the 
proportion x{t) of A- type at time t evolves continuously according to the following stochastic 
differential equation 

dx{t) = [-6x{t) + 6'{1 - x{t))]dt + 2[x{t){l - x{t))]^/^dWt, x{0) = r], (1) 

where (Wt) is a standard one- dimensional Brownian motion and r/ is a random variable 
with values in (0,1) independent of {Wt). This process is known as the Wright-Fisher 
gene frequency diffusion model with mutation effects. It has values in the interval (0, 1). It 
appears as the diffusion approximation of the discrete time and space Wright-Fisher Markov 
chain and is used to model the frequency of an allele A in a population of genes composed 
of two distinct alleles A and a (see e.g. Karlin and Taylor (1981, p. 176-179 and 221-222) 
or Wai-Yuan (2002, Chap. 6)). Suppose now that the current state x{t) cannot be directly 
observed. Instead, at times ti,t2, ■ ■ ■ ,tn with < ti < t2 . ■ ■ < we have observations 
y{ti) such that, given the whole process {x{t)), the random variables y{ti) are independent 
and the conditional distribution of y{ti) only depends on the corresponding state variable 
x(ti). More precisely, we consider the following discrete conditional distributions. Either, a 
binomial distribution, i.e., for AT > 1 an integer. 



P{y{ti) = y\x{u) = x) = {^^^ xy{i - xf-y, y = o, i, . . . , a^, 

or, a negative binomial distribution, i.e., for m > 1 an integer. 



(2) 



P{y{U)=y\x{U)=x)=['^^y %^{l-x)y, y = 0,1,2,.... (3) 



Under these assumptions, the joint process {x{tn), y{tn)) is a hidden Markov model (see e.g. 
Cappe et al, 2005). 

In this context, a central problem that has been the subject of a huge number of contri- 
butions is the problem of filtering, prediction or smoothing, i.e. the study of the conditional 
distributions of x[ti) given y{tn), • • • , y{ti), with I = n (filtering), I = n + \,n + 2, . . . (pre- 
diction), I < n (smoothing). These distributions are generally called filters (respectively 
exact, prediction or marginal smoothing filters). Although they may be calculated recur- 
sively by explicit algorithms, iterations become rapidly intractable and exact formulae are 
difficult to obtain. To overcome this difficulty, authors generally try to find a parametric 
family of distributions on the state space of (x(tn)) {i.e. a family of distributions speci- 
fied by a finite fixed number of real parameters) such that if C{xo) G then, for all n, I, 
C{x{ti)\y{tn), . . . , y{ti)) belongs to J-. For such models, the term finite-dimensional filters is 
usually employed. This situation is illustrated by the linear Gaussian Kalman filter (see e.g. 
Cappe et al. 2005). There are few models satisfying the same properties as the Kalman fil- 
ter: It is rather restrictive to impose a parametric family with a fixed number of parameters 
(see Sawitzki (1981); see also Runggaldier and Spizzichino (2001)). 

Recently, new models where explicit computations are possible and which are not finite- 
dimensional filters have been proposed (see Genon-Catalot (2003), and Genon-Catalot and 



1 



Kessler (2004)). Moreover, in a previous paper (Chaleyat-Maurel and Genon-Catalot, 2006), 
we have introduced the notion of computable filters for the problem of filtering and pre- 
diction. Instead of considering a parametric class T ^ we consider an enlarged class built 
using mixtures of parametric distributions. The conditional distributions are specified by 
a finite number of parameters, but this number may vary according to n, /. Still, filters 
are computable explicitly. We give sufficient conditions on the transition operator of (x(t)) 
and on the conditional distribution of y(tj) given x{ti) to obtain such kind of filters. In the 
present paper, we show that these conditions are satisfied by the model above. Therefore, 
the conditional distributions of filtering and prediction are computable and we give the ex- 
act algorithm leading to these distributions. Moreover, we obtain the marginal smoothing 
distributions which are also given by an explicit and exact algorithm. 

The paper is organized as follows. In Section [21 we briefly recall some properties of the 
Wright-Fisher diffusion. In Section [3l we recall the filtering-prediction algorithm and the 
sufficient conditions of Chaleyat-Maurel and Genon-Catalot (2006) to obtain computable 
filters. Section H] contains our main results. We introduce the class J-f composed of finite 
mixtures of parametric distributions fitted to the model (see (jl6p ). We prove that the 
sufficient conditions hold for this class and give the explicit formulae for the up-dating 
and the prediction operator (Proposition 14. ll Theorem 14.21 Proposition 14. 5p . The result 
concerning the prediction operator is the most difficult part and requires several steps. 
Then, we turn back in more details to the filtering-prediction algorithm (Proposition 14. 6| ). 
Moreover, we give the exact distribution of {y{ti),i = 1, . . . , n) which is also explicit. Hence, 
if 6, 5' are unknown and are to be estimated from the data set {y{ti),i = 1, . . . , n), the exact 
maximum likelihood estimators of these parameters can be computed. At last, in Subsection 
14.41 study marginal smoothing. We recall some classical formulae for computing the 
marginal smoothing distributions. These formulae involve the ffitering distributions, that 
we have obtained in the previous section, and complementary terms that can be computed 
thanks to Theorem 14.21 In the Appendix, some technical proofs and auxiliary results are 
gathered. 



2 Properties of the Wright-Fisher diffusion model. 

In order to exhibit the adequate class of distributions within which the filters evolve, we 
need to recall some elementary properties of model ([1]). The scale density is given by: 

s{x) = exp (-(1/2) r Z^!i±^^ilf^du) = x-''/\l - x)-'l\ X € (0, 1). 
J u{l-u) 

It satifies Jq s{x)dx = oo = s{x)dx if and only if 5 > 2 and 6' > 2, conditions that we 
assume from now on. The speed density is equal to m{x) = x^'^'^~^{l — x)^/^~^, x G (0, 1). 
Therefore, the unique stationary distribution of ^ is the Beta distribution with parameters 
S'/2,6/2 which has density 

= ^T^^T^ (4) 



2 



For simplicity, we assume that the instants of observations are equally spaced with sampling 
interval A, i.e. tn = nA, n > 1. Hence, the process (X„ := x(tn)) is a time-homogeneous 
Markov chain. We denote by p/\{x,x') its transition density and by Pa its transition op- 
erator. The transition density is not explicitly known. However, it has a precise spectral 
expansion (see e.g. Karlin and Taylor, 1981, p. 335-336: Note that 2x{t) — 1 is a Jacobi diffu- 
sion process). The results we obtain below are linked with this spectral expansion although 
we do not use it directly (see the Appendix). 



3 Sufficient conditions for computable filters. 

First, we focus on filtering and prediction and we consider case ([2]) with = 1 for the 
conditional distributions of Yi := y{ti) given Xi = x{ti), i.e., we consider a Bernoulli condi- 
tional distribution. The other cases can be easily deduced afterwards (see the Appendix). 
Let us set 

P{Yi = y\X, = x) = Uy) = xy{l - xf-y , y = 0, 1, x G (0, 1). (5) 

We consider, on the finite set {0, 1}, the dominating measure ^{y) = 1, y = 0, 1. Thus, fx{y) 
is the density of C{Yi\Xi = x) with respect to 



3.1 Conditional distributions for filtering, prediction and statistical in- 
ference. 

Denote by 

vi\^.,^ = C{Xi\Yn,...,Y^), (6) 

the conditional distribution of Xi given (1^, . . . , li). For / = n, this distribution is 
called the optimal or exact filter and is used to estimate the unobserved variable X^ in 
an on-line way. For / = n + 1, the distribution is called the prediction filter and is used 
to predict Xn+i from past values of the 1^'s. For 1 < / < n, it is a marginal smoothing 
distribution and is used to estimate past data or to improve estimates obtained by exact 
filters. 

It is well known that the exact and prediction filters can be obtained recursively (see 
e.g. Cappe et al. (2005)). First, starting with z^i|o:i = C{Xi), we have 

^n\n:l{dx) (X Vn\n-l:l{dx) fx{Yn) ■ (7) 

Hence, 

T^n\n:l = ^Y„{'^n\n-l:l) (8) 

is obtained by the operator ipy with y = Yn where, for v a probability on (0, 1), ^yi^) is 
defined by: 

^y{v){dx) = M^MM, with p,iy) = [ HdOMy)- (9) 
p<y{y) J{o,i) 
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This step is the up-dating step which allows to take into account a new observation. Then, 
we have the prediction step 

Vn+l\n:l{dx') = I Vn\n;l{dx)pA[x , x')dx' = l'n\n;lPA{dx') , (10) 
J{0,1) 

which consists in applying the transition operator: v — > vPi^. These proper- 
ties are obtained using that the joint process (X„,y„) is Markov with transition 
p/^{xn,Xn+i)fxr,+i{yn+i)dxn+il^{dyn+i), and initial distribution Ui\Q.i{dxi)fx^{yi)n{dyi). 
Moreover, the conditional distribution of Yn given (l^n-i, ■ ■ ■ ■, Yi) has a density with respect 
to /X, given by 

Pn\n^l:l{yn) = Pu^U,_^.^{yn) = I T^n\n~l:l{dx) fx{yn) ■ (H) 

J(0,1) 

Note that (Yn) is not Markov and that the above distribution effectively depends on all 
previous variables. For statistical inference based on (Yi,...,!^), the exact likelihood is 
given by 

n 

p4yi,...,y„) = np.,,_i.(>^.)- (12) 
1=1 



3.2 Sufficient conditions for computable filters. 

Now, we recall the sufficient conditions of Chaleyat-Maurel and Genon-Catalot (2006). First, 
consider a parametric class J- = {vq, S 0} of distributions on (0, 1), where G is a parameter 
set included in W, such that: 

• CI: For y = 0, 1, for all u G J^, ^y{v) belongs to i.e. ^y{ve) = ^Ty(e)j ^or some 

Ty{e) G e. 

• C2: For all v ^ vP/\ = '^xi^jyOtxUQ^ is a finite mixture of elements of the class i.e. 

A is a finite set, a = {a\, A G A) is a mixture parameter such that, for all A, > 
and X^AeA ~ 1' G ©> for all A G A. 

Proposition 3.1. Consider now the extended class Tf composed of finite mixtures of dis- 
tributions of T. Then, under (C1)-(C2), the operators (py, y = 0, 1 and v — > z^Pa ore from 
Tf into Tj. Therefore, once starting with vi^q-i = C{Xi) G Tf, all the distributions i'n\n:i 
(exact filters) and Vn+i\n:i (prediction filters) belong to Tf. 

For all n, these distributions are completely specified by their mixture parameter and the 
finite set of distributions involved in the mixture. Of course, the number of components 
may vary along the iterations, but still remains finite. Thus, these distributions are explicit 
and we say that filters are computable. 

The proof of Proposition 13.11 is elementary (see Theorem 2.1, p. 1451, Chaleyat-Maurel 
and Genon-Catalot, 2006, see also Proposition 14.61 below) . Now, it has a true impact because 
the extended class is considerably larger than the initial parametric class. Evidently, the 
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difficulty is to find models satisfying these conditions. Examples are given in Chaleyat- 
Maurel and Genon-Catalot (2006): The hidden Markov process {x{t)) is a radial Ornstein- 
Uhlenbeck process and two cases of conditional distributions of y(tj) given x{ti) are proposed. 
Other examples are given in Genon-Catalot (2003) and Genon-Catalot and Kessler (2004). 
Here, we study a new and completely different model. It has the noteworthy feature that, 
to compute filters, the explicit formula for the transition density of {x{t)) is not required, 
contrary to the examples of the previous papers. (Details on the transition density are given 
in the Appendix). 



4 Main results. 



Our main results consists in exhibiting the proper parametric class of distributions on (0, 1) 
and in checking (C1)-(C2) for this class and the model specified by ([T]) and The interest 
of these conditions is that they can be checked separately. Condition (CI) only concerns 
the conditional distributions of Yi given Xi and the class J^. In the Appendix, we prove 
condition (CI) for the model specified by ([T]) and ([2]) or (l3|). Condition (C2) concerns the 
transition operator of {x{t)). It is the most difficult part. Let us define the following class 
of distributions indexed by B = N x N: 

^ = {j^i,j{dx) oc hij{x)Tr{x)dx, G N x N}, (13) 

where 

hij{x) = x'{l-xy. (14) 

Hence, each distribution in is a Beta distribution with parameters (i + j + f ) and (see 

(SD) 

Vij{dx) = - ^-77 — \(n^\{x)dx. (15) 

''^ ' S(i + I,j + I) ' ^ ' 

Let us define the extended class: 

= i 5Z ^ N X N, |A| < +oo,a = (a^j, (i, j) G A) G 5/}, (16) 

(«j)eA 

where 

5/ = {a = (ay,(i,j)GA),AcNxN,|A|<+oo,V(^,i),aij>0, ^ = 1} (17) 

is the set of finite mixture parameters. It is worth noting that the stationary distribution 
■K{x)dx = VQfi{dx) belongs to T. Thus, in the important case where the initial distribution, 
i.e. the distribution of r] (see ([U), is the stationary distribution, the exact and optimal 
filters have an explicit formula. 
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4.1 Condition (CI): Conjugacy. 



Proposition 4.1. Let Uij G T (see ifT^)). 

1. For y = 0,1, ipy{vij) = Hence, (CI) holds. 

2. The marginal distribution is given by 

I y( 



Proof. The first point is obtained using tiiat fx{y)i'ij{dx) oc x'^^y^~^^{l — xy^^^y^2^^dx oc 
/ii+yj+i_j;(x)7r(x)dx. 

For tiie marginal distribution, we iiave 



/ J I /, -|- // -|- 



5(i + y + |,j + l-2/ + 2. 



To get (dHI), we use the classical relations B{a, b) = ^■^'{(^b) ^'^'^ + 1) = aT{a), a,b > 
where r(.) is the standard Gamma function. □ 

Proposition 14. 1 1 is proved for the conditional distributions ([2]) and ([3|) in the Appendix. 

Remark : Proposition 14.11 states that the class .F is a conjugate class for the parametric 
family of distributions on {0, 1}: y fx{y)lJ'{dy) with respect to the parameter x € (0, 1). 
The property that Beta distributions are conjugate with respect to Bernoulli distributions 
is well known in Bayesian statistics (see e.g. West and Harrison (1997)). Analogously, the 
class T is also a conjugate class for the parametric family of distributions ([2]) and ([3]) with 
respect to x € (0, 1). <0 



4.2 Condition (C2): Introducing mixtures. 

The class is the natural class of distributions to consider for the conditional distributions 
([5]) because this class contains the stationary distribution ■K{x)dx = UQfi{dx) of ([1]). We do 
have = z^o,o- However, when {i,j) ^ (0,0), i'ijPa no more belongs to T but belongs 

to as we prove below. We need some preliminary properties. 

Proposition 4.2. For all Borel positive function h defined on (0, 1), ifi'{dx) = h{x)'K{x)dx, 
then, vPt{dx') = Pth{x')iT{x')dx' , where, fort > 0, 

Pth{x') = I h{x)pt{x' ,x)dx. 
Jo 
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Proof. It is well known that one-dimensional diffusion processes are reversible with respect 
to their speed density: The transition pt{x,x') is reversible with respect to TT{x)dx, i.e. 
satisfies for all {x,x') G (0, 1) x (0, 1): 

7r{x)pt{x,x') = Tr{x')pt{x',x). (19) 

This gives the result. □ 

Proposition 4.3. Suppose that: 

• (C3) For all {n,p) G N x N, there exists a set A„_p C N x N such that |A„^p| < +oo 
and for all t >0, 

Pthn,p{-)= Yl B^At)hiA-)^ (20) 

{i,j)eA„,p 

with, for all {i,j), and all t > 0, Bi j{t) > 0. 

Then, condition (C2) holds for Pt for all t >0. Moreover, 

i^n,pPt{dx) = ^ aij{t)uij{dx), 

(ij)6An,p 



where a{t) = {aij{t), {i, j) E A„^p) belongs to Sj (see (T7^) and 



a,,{t) = B,,{t) f'^y'^'^] (21) 



Proof. We have (see ([TO]) ), for all(n,p), 



i?(n+|,p+|)' 



Using Proposition I4.2t we get: 

{hn,p1T)Pt{dx) = Pthn,p{x)-K{x)dx. 

Now, 

PthnA-X) = E B,^,{t)h,,j{.)A.), 

(«,i)GA„,p 

and ^ 

hi^j{x)T:{x)dx = '^id(dx) ^^^ ^^J^^^-^^J'^^^'^^ . 

Joining all formulae, we get the result. □ 
Therefore, it remains to prove condition (C3). We start with a classical lemma. 
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Lemma 4.1. Let h belong to the set C^((0, 1)) of bounded and twice continuously differen- 
tiable functions on (0,1). Then, 

^(Pth{x)) = PtLh{x),Poh{x) = h{x), (22) 
at 

where Lh{x) = 2x{l — x)h"{x) + [—5x + 5'{\ — x)]h'{x) is the infinitesimal generator of ([7p. 

Proof. Let Xx{t) be the solution of ([1]) with initial condition Xx{^) = x. By the Ito formula, 

h{xx{t)) = h{x) + / Lh{xx{s))ds + 2 / h'{xx{s)){xx{s){l-Xx{s))f''^dWs. 
Jo Jo 

By the assumption on h, taking expectations yields: 

Pth{x) = h{x) + j PsLh{x)ds, 
Jo 

which is the result. □ 

Now, we start to compute Pthn,p{x) for all n,p G N. 
Proposition 4.4. Let mn^p{t, .) = Pthn,p{-) Then, for all n,p G N, 

^mn,pit, .) = -an+pmn,p{t, .) + Cni5')mn-i,pit, .) + Cp{6)mn,p-i{t, .), mo,o(i, •) = 1, (23) 

where, for all n G N, 

an = n[2{n - 1) + 5 + 6'], Cn{5) = n[2{n - 1) + 6]. (24) 

(If p or n is equal to 0, then, Cp{6) = or c„((5') = 0, and formula holds). Note that, 
since the expression of On is symetric with respect to 5, 5' , we do not mention the dependance 
on these parameters. Note also that for all n, since both 6 and 6' are positive ( actually >2), 
the coefficients On and Cn{6), Cn{6') are non negative. 



Proof. In view of (j22p . it is enough to prove that 

Lhn^p = —On+phn^p + Cn{S )hn-l^p + Cp{5)hn^p-l, (^5) 

where L is defined in Lemma 14.11 To make the proof clear, let us start with computing 
Lhnfl. We have immediately: 

Lhnfi{x) = -anhnflix) + Cn{6')hn^lfi{x) . 

Now, since y{t) = 1 — x(t) satifies 

dyit) = [-6'y{t) + 6(1 - y(t))]dt + 2(y(t)(l - yit))'/^dWt, (26) 

we obtain L/io,p by simply interchanging 5 and 6' and get 

LhQ,p{x) = -apho^p{x) + Cp((5)/io,p-i(x). 

Finally, to compute Lhn,p, we use the following tricks: Each time x""*"^ appears, we write 
x'^~^^ = — (1 — X — l)x" = —{l — x)x'^+x'^\ each time (1 — x)^"*"^ appears, we write (l — a;)^^^ = 
(1 — xY{l — x) = (1 — xY — x(l — xY . Grouping terms, we get ([25]) . □ 
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Our aim is now to prove that 
mn,p{t, .) = eyip {-an+pt)hn,p{.) + ^ B'^'^^^^__^{t)hn-^k,p-i{-)-, (27) 

0<k<nfl<l<p,{k,l)^(Qfl) 

where, for all (fc, {n,p), B^'^j^ ^_i^{t) > for all t > 0. Moreover, we give below the precise 
formula for these coefficients. Hence, the set A„_p of (C3) is equal to {{k,l),0 < k < n,0 < 
I < p}. The first term can also be denoted by 

Klpit) =exp(-a„+pt). 
It has a special role because it is immediately obtained by ([23|) . 

4.2.1 Computation of mnfi{t,.) and mg, 

Recall notation (jl4p and that mnfi{t, .) = Pthnfi{-)- We prove now that (|27p holds for all 
(n, 0) and all (0, n). 

Theorem 4.1. The following holds: 

n 

mn,o{t,.) = eyip {-ant) hn,o{-) + ^ ^^f^.o (*)^"-fc.o(-)) (28) 

k=l 

where, for all {k, n), with 1 < k < n, B^'^^^ ^{t) > for allt > 0. Moreover, for k = 1, . . . ,n, 

^n-fc,o(*) = Cn{S')Cn-l{S') . . . C^-fc+l (5')^t (^n, On-l , • • • ,On-fc), (29) 

where 

Bt{an,an-i,...,an-k) = (-1)'' X^exp (-On-jt)^^^ ■ 1. (30) 

j=0 ViQ<l<k,li^j\'^ri-j a,n-l\ 

We can also set 

Kfii^) = exp(-a„t). 

Analogously: 

n 

mo^nit, .) = exp (-a„t)/io,n(-) + ^ 5o'n-fc(*)^0,n-fc(-)> (31) 

k=l 

where, for all {k, n), with 1 < k < n, BQ'^_^{t) > for all t > 0. Moreover, for k = 1, . . . , n, 

^o'n-k(^) = C„((5)c„_i(5) . . . Cn^k+l{^)Bt{a 

We also set 

K',ni^) = exp(-a„t). 
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Proof. For the proof, let us fix x and set m„^o(^) ^) = f^nit)- We also set B^'^^^ ^(t) = B^_^{t) 
during this proof. Solving m'n{t) = —anmn{t) + c„(5')m„_i(t), m„(0) = = hnfi{x) yields 

mn{t) = exp {—ant) + exp (— a„t) / exp (a„s)m„_i(s)(is. (32) 
Let us first prove by induction that 

n 

mn{t) = Y,K-k{t)x''-\ (33) 

fc=0 

where B'^_,^{t) > for ah t > and ah = 0, . . . , n and B^^H) = exp (-a„t). For n = 0, 
mo{t) = 1. For n = 1, we solve (j32]) and get 

mi(t) = exp (-ait)x + ^^(^/) (1 " ^^P (""i^)) ^ 

ai 

So, (j33p holds for n = 1 with 

=exp(-ait), i?i(t) = ci(5')^i^^^^^^^^>0. (34) 

Suppose ([33]) holds for n — 1. We now apply ([32]) . Identifying the coefficients of x"^'^, 
< k < n, we get: 

Bli{t) = exp {-ant), 

and for /c = 0, 1, . . . n — 1, 

= cn(5') exp i-ant) exp (a„s)i?;:zi_,(s)(is. (35) 
Hence, ()33p holds for all n > with all coefficients non negative. 



Now, we prove (j29|) - (j30|) by induction using (j35|) . For n = 1, we look at (jMj) and see 
that, since ao = 0, 

Rl^+^ ^^^ exp(-ai^) (-1) exp (-apt) /c/nr/ n 

-Do(^) = ci(c> )(-!)[ \ . . J = ci{d )Bt{ai,ao). 

ai — ao \ao — ai\ 

Now, suppose we have formulae (|29 p - (|30p for n — 1 and /c = 0, 1, . . . , n — 1. We know that 
Bn{t) = exp {-Ont). Let us compute, for A; = 0, 1, . . . , n — 1, using (f35]) . We 

have: 

^n-(fc+l)(0 = Cn((5')Cn-l((5') • • • C„_fc (5' ) (- 1)^' X B, (36) 

with 

5 = ^ exp (-a„t) / exp ((a„ - a„_i_j)s)(is= ■ -. (37) 

Integrating, we get: 

{—ly 

B = ^exp (-a„_i„jt)- \ r + (- exp (-a„t))^. 
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with 

^ = E7 ,n I r. (38) 

Hence, 

5 = ^ exp (-a„_j/t)= ^ 1 + (-exp (-a„t))A. (39) 

j/^l \.\.0<l'<k+l,l'^j' l^n-j' Cln-l'l 



In view of (ISgp-ilM])-®-®-®, to complete the proof of ([28]), it remains to show the 
following equality: 



Lemma 4.2. 

k 



y t}l ^ I , 

^ (a„ — Un-l-j) no</<fc,/^j I'^n-l-j ~ 0,n-l-l\ («n. — an-l)(«n — an-2) • • • (^n — On-fc-l) 



This lemma requires some algebra and its proof is postponed to the Appendix. At last, 
to get ()3ip . we just interchange 6' and 5 in all formulae because of ()26p . □ 

4.2.2 Computation of m„,p(t, .). 

Recall that hn,p{x) = x'^{l — xY and mn,p{t, .) = Pthn,p{-)- Now, we focus on formula (p3|) . 
It is easy to see that, since we have computed rrinflit, .) for all n and mo,p(t, •) for all p, 
then, we deduce from ([23l) mn^p{t, .) for all {n,p). This is done as follows. Suppose we have 
computed all terms mij-i{t, .) for < « < j < n, then, we obtain all terms mij-i{t, .) for 
0<i<j<?T- + l. Indeed, the extra terms are: 

• m„4.i^o(^) •) that we know already, 

• mj^„+i_j(t, .) for < i < n + 1 that is calculated from 

^mi^n+i-i{t, .) = -a„+imj,„+i_j(t, .)+Ci{d')mi^i^n+i-i{t, .)+Cn+i-i{S)mi^n+i-i-i{t, ■), 



• at last, uiQ^n+iit, .) that we know already. 



This is exactly filling in a matrix composed of the terms m„^p(t, .). Having the first line 
TTT'0,n{t^ ■) and the first column mn,o{t, .), we get each new term mij{t, .) from the one above 
(mj_ij(t, .)) and the one on the left {rriij-iit, .)). 

Now, we proceed to get formula (|27p . 
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Theorem 4.2. For all such that < i < j < n, 

mij-i{t, .) = exp {-ajt)hij-i{.) + ^ Bl'lkj-i-i(''^)^i~k,j^i-i{-), (40) 

0<k<i,0<l<j-i,{k,l)^{Ofl) 

with 

■ ■-■ fk + l\ 

Bl'\j^i^i{t) = ( ^ jQ((50.--c,_fc+i((50cj-i(5)...Cj_i_i+i((5)St(aj,aj_i,...,aj_(fc+;)), 

with the convention that, for k = 0, there is no term in c. (5') and for I = 0, there is no term 
in c ((5) )■ 

Proof By ([291) -([30]) -([35]), we have proved that 

Bt{an+i,an,...,an-k) = ex.p{-an+it) / exp (a„+is)Ss(a„, a„_i, . . . , a„_fc)ds. (41) 

Jo 

Suppose (jlO]) holds for < i < j < n. Let us compute the extra terms mi^n+i-i{t, .) for 
< i < n + 1 using their differential equations. We have 

mi^n+i-i{t, .) = exp (-a„+ii)/ii,n+i-i(-) + Ai{6') + Bi{6), (42) 

with ^ 

Ai{5') = exp {-ttn+it) / exp (a„+is) Q((5')"ii_i,„+i_i(s, .)ds, (43) 

JO 

Bi{S) = exp {-an+it) exp {un+is) Cn+i-i{S)mi^n+i-i-i{s, .)ds. (44) 

JO 

We apply the induction formula and replace rui-i^n+i-iis, .),mi^n+i-i-i{s, .) by their devel- 
opment. This yields: 

mi_i,„+i_j(s, .) = exp (-a„s)/ii_i,„+i_i(.) 

+ Bl^l'^f!'^^^-^^_._i,{s)hi_i_k',n+i-i-i'{-), 

0<fc'<i-l,0<Z'<ri+l-i,(fc',i')^(0,0) 

mi,n+i-i-i{s, .) = exp (-a„s)/ii,„+i_i_i(.) 

+ X] Bl'^^}~l~l^^^^i,,{s)hi^k'',n+i-i-i-i''{-)- 

0<k"<i,0<l"<n+l-i-l,{k",l")¥={0,0) 

In (jl2]) - (P3]) - (fil]) . the coefficient of hi-i^n+i-ii-) obtained by the above relations only comes 
from: ^ 

Ci{S') exjp {-ttn+it) / exp (a„+is) exp (-ans)ds. 

JO 

By (HT]l . this term is equal to: 

Ci(5')^t(«n+l, a„) = 5-:"i"['^+l_i_o = (J) Ci((5')5t(an+1, On)- (45) 
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Analogously, the coefficient of hi^n+i-i-i{-) comes from: 

Cn+i-i{d) e^p {-Qn+it) / exp (a„+is) exp (-ans)(is. 

^0 

This term is equal to: 

Cn+l-iiS)Bt{an+l,an) = Bl'^'^;^lZl^i = (^^Cn+l-i{S)Bt{an+l,an). (46) 

Now, the coefficient of the current term hi-k^n+i-i-l{-) comes from the sum of the following 
two terms: 

bi = Ci{6')eicp{-an+it) j exp (a„+is)5'~J:^(+^"^^ ,^^^_._^(s)(is, (47) 

{i — 1 — k' = i — k,n + 1 — i — I' = n + 1 — i — I, thus k' = k — 1,1' = I) and 

b2 = Cn+i-i{6) exp (-ttn+it) exp (a„+is)S*f +^"*l^._^(s)ds, (48) 

Jo 

(i - k" = i - k,n + 1 - i - 1 - I" = n + 1 - i - I, thus k" = k,l" = 1-1). Thus, 
bi = Ci{S')ex.p{-an+it) J exp(an+is)( Jci_i((5') . . . Ci_i_(fe_i)+i((5')x 

Cn+l-i{S) ■ ■ ■ Cn+l-i-l+l{S)Bs{an, On-l, ■ ■ ■ > «n-(fe+/-l))<^^ 
^ Ci((5')Ci_i((5') . . . Ci-k+l{S')Cn+l-i{S) . . . Cn+l-i-l+l{S) 

X Bt{an+l,an, fln-l) • • • > 0'n+l-{k+l))- 

b2 = Cn+i-iid) exp {-ttn+it) J exp(a„+is)( ^ J Cj(y) ■ ■ ■ Cj_fe+i((5')x 

((5)5, (a 

= """^ Ci{d')Ci-i{6') . . . Ci-k+l{S')Cn+l-i{S) . . . Cn+l-i-l+l{S) 

X Bt{an+1, an, Qn-l, ■ ■ ■ , a„+i_(fc+/)). 

Now using that C^^l^^) + C'^i'^) = Ct'^)^ "^^ finally obtain that hi + 62 is exactly equal to 
the expected term B^'_i^^^_^^_j^_i it). □ 



And 
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4.2.3 Back to the mixture coefficients. 



Now, we use Proposition 14.31 and formula (j2ip to obtain the mixture coefficients for fjj_jPt 
We begin with a lemma. 



Lemma 4.3. For < i < n and < j < p, we have 

(49) 



B{i + f-, j + |) (*^'')an+pan+p-i • • • ai+i+pO-i+pai+p-i ■ ■ ■ fli+j+i 



i?(n + f ,p + I) ("+P)c„(5')cn-i(<5') . . . Ci+i{5')cp{5)cp.i{5) . . . cj+i{6) ' 

In the trivial case {i,j) = {n,p), the quotient is equal to 1. In for j = p, there is no 

term in c. (5), and for i = n, there is no term in c,{5'). 

Proof. We use the relation 
Hence, 

B{n + -,p+-) = —— g,^^ B{n-l + -,p-l + -). 

By ([Ml), we have: 

an -, , '^' + '^ c„(5') y Cp((5) <5 

— =n — IH , =n— IH , — =n— IH — . 

2n 2 ' 2n 2 ' 2p ^ 2 

Hence, 

2 2 In Ip Un+p fln+p-i 2 2 
Iterating downwards yields: 

-^(^ + j + I) _ _(^^K7)[n!p! an+jan+j-i ■ ■ ■ a^+j+l 

+ + ~ {n + p)\ i! j!c„((5')c„_i((5')...Ci+i((5')cp((^)cp_i((^)...Cj+i(5)' 

This gives (gS]). □ 



Now, we have the complete formula for Uij-iPt. 
Proposition 4.5. For < i < j, we have 



^iJ-iPt = al'ikj-i-ii^) ''i-kj-i-i, 

k=0,...,i,l=0,...,j-i 



where, for {k,l) ^ (0,0), 



a 



i—k,j—i—l 



For {k,l) = (0,0), 



(t) = . ajaj-i . . .aj_k-i+iBt{aj,aj_i, . . . ,aj_k~i)- (51) 

[k+U 

ay::(t) = exp(-a,t). (52) 
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The proof is straightforward using Proposition 14.31 Theorem 14.21 and the lemma. 

Let us now make some remarks concerning the above result. First, note that the mixture 
coefficients are symetric with respect to 6' and 6. The non symetric part appears in the 
distributions Vi~k,j-i-i- Another point is that, looking at Vij^iPt, we see that very few 
mixture coefficients will be significantly non nul. Indeed, they are all composed of sums of 
rapidly decaying exponentials. 

To illustrate our result, let us compute more precisely some terms, e.g. i^ifl, 1^2,0- For 
n = 1, ai'o(t) = exp {—ait), aj' o(t) = aiBt{ai,ao) = 1 — exp {—ait) and ai = 5' + 5, oq = 0. 
Hence: 

1/1,0 = exp {-{6' + 5)t)uifl + (1 - exp {-{5' + 5)t))fo,o- 

For n = 2, 

«2;o(0 = exp(-a2t), 

Q^Q(t) = a2Bt{a2,ai) = — — — (exp (-oit) - exp (-02*)), 

02 — ai 

Q^n'nl*) = a2aiBt{a2,ai,ao) = exp (-02*) exp(-aii) + l, 

a2 - ai 02 - ai 

with a2 = 2(2 + 5' + 5), ai = 5' + 5, = 0. And so on ... . 

Note also that Proposition 14.51 and result (I40p can be explained by spectral properties 
of the transition operator Pt. Indeed, considered as an operator on the space L^(7r(x)(ix), it 
has a sequence of eigenvalues and an orthonormal basis of eigenfunctions. The eigenvalues 
are exactly the (exp (— a„t), n > 0). The eigenfunction associated with exp(— a^i) is a 
polynomial of degree n, linked with the n-th Jacobi polynomial with indexes (^ — 1) f — 1) 
(see the Appendix). Thus, each polynomial hij has a finite expansion on this eigenfunctions 
basis. Therefore, Pthij has also a finite expansion on the same basis. However, from these 
spectral properties, it is not evident to guess the expansion obtained in (jlUj) nor is it to 
guess that the expansion contains only positive terms that lead to mixture coefficients. 

Finally, Proposition |45] shows that, for all t>0, J2o<k<i,o<i<j-i'^l-kj-i-ii'^) — ^- '^^^^ 
can be checked directly by formulae ([?T|) -(|52 p (see the Appendix). 

4.3 Working the filtering-prediction algorithm and estimating unknown 
parameters. 

We must now illustrate how Proposition 13.11 allows to obtain explicitly the successive dis- 
tributions of filtering i/„|„:i and of (one-step) prediction i^n+i\n:i (see Suppose that 
the initial distribution is C{Xi) = fo,Oi i-s. the stationary distribution of {x{t)). After 
one observation Yi, we have the up-dated distribution vi^i-i = ^^1(^^0,0) = '^Yi,i-Yi- Then, 
we apply the prediction step to get ;/2|i:i = t^Yi,i-YiPA- This distribution is obtained by 
Proposition 14.51 

l^Yr,l-YiPA = Yl "Yi-kJ-Yi-li^)''yi-k,l-Yi-l- (53) 

o<fc<yi,o<i<i-yi 
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Then, there is another up-dating for Y2, and another prediction, and so on. To be more 
precise, let us state a proposition that explains the use of Proposition I3.1[ 

Proposition 4.6. Suppose v = J2o<k<io<i<j '^i-kj-i ^i-k,j-i is a distribution of Tf. 

1. Then, for y = 0, 1, 

^y{^)^ (^i-k,j-lPu,_^^^_i{y)^i+y-k,j+l-y-U 

0<k<i,0<l<j 

where the marginal distribution p^-_^ ._i^{y) is given in [T8\}. Thus, 

^y{t^) = Cti+y-kJ+l-y-l Vi+y-k,j+l-y-li 

0<k<i+y,0<l<j+l-y 

where Ui+y-kj+i-y-i oc Oi-kj-i Pv,_k.j^i{y) for k = 0,1,..., i,/ = 0,1,..., j and 
ai^y-k,j+i-y-i = otherwise. 

2. We have: 

0<K<t,0<A<j yO<fc<K,0<KA J 

where the a^Z'^'^jZ^i^) o,re given in Proposition \4-5\ 

3. The marginal distribution associated with v is 

Pv{y)= Y <^i-k,j-iPui_k,j-iiy)- (54) 

0<k<i,0<l<j 

It is therefore a mixture of Bernoulli distribution (see Proposition \4-l\ and formula 

The first part is straightforward. The second part is an application of Proposition 14.51 
with an interchange of sums. Thus, the number of components in the successive mixture 
distributions grows. Indeed, let us compute the number of mixture components for the 
filtering distributions. For we find (1-|-Yi)(l-|-1 — Yi); the prediction step preserves 

the number of components. For Vnln-.i and i^„+i|n:i) the number of components is (1 + 
^^^i Yi){l + n — Yl^=i However, as noted above, very few mixture coefficients will be 
significantly non nul. It was also the case for the model investigated in Genon-Catalot and 
Kessler (2004). 

Let us notice that the /i-step ahead predictive distribution, i'n+h\n:i is obtained from 
^n\n:i by applying the operator P^, i.e. i'n+h\n:i = ^n\n:iPA- Therefore, this distribution 
stays in the class and has the same number of mixture components as Vn\n:i- 

Now, suppose that 6' , 5 are unknown and that we wish to estimate these parameters using 
the data set {Yi, . . . , y„). The classical statistical approach is to compute the corresponding 
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maximum likelihood estimators. This requires the computation of the exact joint density 
of this data set which gives the likelihood function (see (jl2p ). For general hidden Markov 
models, the exact formula of this density is difficult to handle since the integrals giving the 
conditional densities of Yi given (l^-i, . . . , Yi) are not explicitly computable (see formula 
pip ). On the contrary, in our model, these integrals are computable by formula (|18p . 
Suppose that the initial distribution is the stationary distribution of ([T|), i.e. fo,o- For 
i = 1, the law of Yi has density Puofiiui) given by ([TS]) : It is a Bernoulli distribution with 
parameter jrpj- Then, for i > 2, the conditional distribution of Yi given Yi-i, . . . , Yi has 
density py.^._^^^{yi). It is now a mixture of Bernoulli distributions. The exact likelihood is 
therefore a product of mixtures of Bernoulli distributions. 



4.4 Marginal smoothing. 



In this section, we compute vi\n-i for I < n. To simplify notations, denote by p{xi\yn-, ■ ■ ■ , yi) 
the conditional density of Xi given Yn = y„, . . . , Yi = yi, i.e. the density of i'i\n:i taken at 
Yn = Vni ■ ■ ■ ,Yi = yi- Analogously, denote by p{yi\yi-i, . . . , yi) the conditional density of 
Yi given Yi^i = yi-i, . . . , Yi = yi. We introduce the backward function: 

pi,n{yi+i^--- ^yn;x), (55) 

equal to the conditional density of (Yj+i, . . . , Y^) given Xi = x. By convention, we set 
Pn^ni^'ix) = 1. Then, the following forward-backward decomposition holds. 

Proposition 4.7. For I < n, 

I I X p{xi\yi,---,yi) , . 

P[Xi\yn, - ■ ■ ,yi) = ffn 7—\ sPl,n[yi+l'---^yn;Xl) (56) 

Ui=i+iPiyi\yi-ii---,yi) 



This result is classical and may be found e.g. in Cappe et al. (2005). Therefore, the 
smoothing density is obtained using the filtering density that we have already computed. 
The denominator in (j56p is also available. It remains to have a more explicit expression 
for the backward function (j55p . The following proposition gives a backward recursion from 
I = n — \ down to Z = 1 for computing ()55p . 

Proposition 4.8. First, for all n, 

Pn-l,n{yn]x) = P/\[f .{yn)]{x) ■ (57) 

Then, for I + 1 < n, 

Pi.n,{yi+i, ...,yn;x) = PA[f.{yi+i)pi+i,n{yi+2, ■■■,yn; -^x) (58) 



Proof. We use the fact that [XmYn) is Markov with transition PA{xmXn+i) fx^+iiVn+i) 
Given Xn-i = x, Xn has distribution p/^{x, Xn)dxn- Hence, 

Pn-l,niyn; X) = / PAix , Xn) fx„{yn)dXn, 

Jo 
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which gives (j57p . Then, for n > 1 + 2, 



Pi,n{yi+i^ ■ ■ ■ 



lUn^x) 



PA(x,x/+i)/;:c,+i(y/+i) X JJ pA{xi-i,Xi)fx,{yi)dxi+i. . .dx, 

1=1+2 



■n 



L 



1 



PA{x,xi+i)fxi^-^{yi+i)pi+i^n{yi+2-, ■ ■ . ,yn;xi+i)dxi+i, 



which gives (j58p . 



□ 



Let us now apply these formulae to our model. We will show briefly that backward 
functions can be computed by simple application of Theorem 14.21 Indeed, since 

Pn-l,n{yn; X) = PAhy„,l-y^{x) = my^^i^y^{A, x) , 

is obtained by Theorem 14. 2[ Next, we compute 

'^yuA-yni^^ ■) X V-i,i-s/n-i(-)> 

which is a linear combination of hy^_-^^y^_i^^2-y„-i-yn-i with < k < yn, < / < 1 — y„ and 
apply the transition operator Pa to get Pn-2,n{yn-i-, yn'-, x). This is again given by Theorem 
14. 2[ By elementary induction, we see that backward functions are explicit. 
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5 Appendix 

5.1 Proof of Lemma 14.21 

Let us write in more details expression (j38p . We have 
1 1 



A 



(a„ — a„_i) (ctn-l — an-2)(«ri-l — fln-s) • • • (fIn-1 " Ctn-l-fc) 
(-1) 1 

{an — a?i-2)(an-l — an-2) (an-2 — fln-s) (an-2 — ^n-l-fc) 



+ 



(flfj 0,^— j)(fln— 1 . . . ((2^_j_)^i Ofi—j) {Ofi—j Q^n— J — 1) • • • {(^n—j C^ra— fc— l) 

(-1)' 



(fln — an.-l-A:)(an-l — «n-l-A;) (an-2 — «n.-l-A:) • • • (On-fc — ^n-l-fc) 

Now, we set 

Lq = (a„ — a„_i)(a„ — a„_2)(a„ — a„_3) . . . (a„ — a„_fc_i), 
Li = (a„_i — a„_2)(an-i — fln-s) (On-l — an-fe-l), 

L2 = (Oji-2 — 0?i-3)(«n-2 — an-4) • • • («n-2 — fln-fc-l), 

Lk-l = (On-fc+l — a-n-k){0'n-k+l — ara-fc-l), 

Lk = {cLn-k — dn-k-l)- 

We must prove that 

A = ^. (59) 
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For this, we introduce the product T„ = LqLi . . . L^. Now, we need to prove that AT^ = 
L1L2 • • • Lk- We start to compute AT„: 

ATn = L2 ■ ■ ■ Lk 

+ 7 777 T-f^S ...Lk + ... 

[an — an-2){{an-l — O-n-2) 

+ 7 77 r -, x^j+l ■ ■ ■ i^k + ■ ■ ■ 

_l ( — l)^-Lo-Li ■ . . Lfc_l 

(a„ — a„_A,._i)(a„_i — a„_fe_i) . . . {an-k+i — On-fc-i) 

Now, we see that 

ATn = P{an) (60) 

where P{.) is a polynomial with degree k. Indeed, in ^T„, the terms containing a„ come 
only from the terms 

7 = Pj(a„), 

where 

Pl{x) = {x- an-2){x - Qn-i) . . . (x - a„_fc_i), 

Pj{x) = {x - a„_i)(x - an-2) ■■■{x- fln-i+i) X (x - an-j-i) ...{x- a„_fc_i), 

Pk+i{x) = {x- an-i){x - an-2) . . . (x - a„_fc), 

are all products of k factors of degree 1. Notice that Pj{x) is nul for x = 
flra-i! «n-2, ■ ■ ■ , fln-j+i, «n-j-i, • • • , On-fc-i and that Pi(a„_i) = Li, 

Pji^n-j) = {an-j — an-l){an-j — an-2) • • • {o-n-j — (l-n-j+l) X Lj, 
Pk+l{an~k-l) = (On-fc-l — an-l)(an-A:-l — 0-11-2) ■ ■ ■ {o-n-k-l — dn-k)- 

Therefore (see (f60]l ) 

P(x) = Pi{x)L2L3 . . . Lk 

-P2{X) ^ L3 . . . Lfc + . . . 

«n-l — an-2 

+(-l)^-^^'.(x)7 w ^1^2. ..Yi ^^,^.1 . . . + . . . 

ia^-l — an-j ) [an-2 — a-n-j) ■ ■ ■ [O-n-j+l — 0.n-j ) 

+{-l)'Pk-,i{x)- L,...Lk-, 



(On-l — a„_i_fc)(a„_2 — an-l-k) ■ ■ ■ {^n-k+l — CLn-l-k) 
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Now, 



-P(an-i) = Pi{an-i)L2L3 . . . Lk = L1L2L3 . . . Lk 

-r(,a„_2j — — -r2(,an-2j7 T-i^S • • • -t^fc = " 7 s^l^^2---^k 

= L1L2 . . . Lk 

L1L 2 . . . Lj-i 
1-2 - 

(-l)20-i)LiL2...L,-iL,L,+i...Lfc 



P(a„_,) = (-ip-ip,.(a„_,)- rL,+i...Lfc 

— Cn-j j[an-2 — O-n-j ) ■ ■ ■ [O-n-j+l — ^n-j ) 



Li . . . Lk-i 



— Cln-l-k){0'n-2 — On-l-fc) • • • (^n-fe+l ~ «n-l-A;) 

= (-l)2%L2...Lfc. 

Therefore, P{x) = L1L2 ■ ■ ■ Lk for the k + 1 distinct values x = a„_i, 0^-2, • • • , On-fc-i- 
Since P{x) is a polynomial of degree k, it is constant equal to L1L2 ■ ■ ■ Lk- In particular, 

P(a„) = L1L2 . . . -Lfc, 

which is equivalent to A = 1/Lq (see (i59]l ). So the proof of Lemma |12] is complete. 

5.2 Binomial or negative binomial conditional distributions. 

Proposition 14.11 holds for the other cases given in the introduction. Let z/jj belong to J-". 

• If /:c(y) = {^)xy{l-x)^~y, y = 0,. . . ,N, then ipy{i^i,j) = i^i+yj+N-y and the marginal 
distribution is equal to: 



N\ B[i + y + [5'/2),j+N-y + {5/2)) 



If fx{y) = ('"+^^->™(l-x)^ y = 0, . . ., then ipy{u,j) = Ui+mj+y and for y = 0, 1, . . . , 
, , _ fm + y-l\ B{i + m + {6'/2),j + y + {6/2)) 



5.3 Mixture coefficients. 

We now check using formula (fSTj) that Ylo<k<i,o<l<j-i'^l-k^j-i-ii^) — 1- interchanging 
sums and setting k' = k,l' = k + I, we first get 

j 

0<k<ifl<l<j-i l'=0 
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where 

/ • . ik') il'-k') 
p{hJ-^)= 2^ 7J-. — • 

0<k'<ifi<l'~k'<j~i 

We recognize the sum of hypergeometric probabihties so that p{i,j — i) = l. There remains 
to prove that, for all i > 0, 

i 

o<fc<j fc=o 
We fix i. Looking at (j30p and interchanging sums, we have to check that 

i 

Hi^j exp {-ai-jt) = 1, 

j=0 



I 



where, for j = 0, 1, . . . , i, 

Hi-j=T.^i (63) 

k=j 

and 

^k — \ n U r, I ' 

Since oq = and i^o = (— . . . ai/oj . . . ai = 1, we have i/o exp (— oot) = 1. So we must 
prove that, for all j = 0, 1, . . . , i — 1, Hi_j = 0. Denote by -D^ the denominator of L|.: 



It is easy to prove by induction (on A;) that, for A; = z — 1, . . . , j + 1, 

^k-i 

where the notation ? means that the term is absent. The formula for k = j + \ yields 
This gives -ffj-j = (see ([63]) ) for all j = 0, 1, . . . , i — 1.. 



5.4 Spectral approach. 

The transition density pt{x,y) of ([T]) can be expressed using the spectral decomposition of 
the operator Pf. Consider equation ([1]) and set z{t) = 2x{t) — 1. Then, 

dz{t) = [-6(1 + z{t)) + (5'(1 - z{t)]dt + (1 - z^{t))'^/^dWt. 
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This is a Jacobi diffusion process. Let us set 

« = ^-l,/5 = f -1. (64) 
Then, for n > 0, u{z) = Pn'^{z) with 

Pn^^) = - + - + ^r^\ (65) 

is solution of 

(1 - z'^)u" + [(3 -a- {a + (3 + 2)z]u' = -n{n + a + /3 + l)u. (66) 

The function (I65[) is the Jacobi polynomial of degree n with indexes (a, /3). The sequence 
{Pn'^ {z),n > 0) is an orthogonal family with respect to the weight function p{z) = (1 — 
z)"(l+z)^l(_i_+i)(z). After normalization, it constitutes an orthonormal basis of L'^{p{z)dz) 
(see e.g. Lebedev (1972, p. 96-97) or Nikiforov and Ouvarov (1983, p. 37)). Now, we set 
h{x) = u{2x — 1) in (i66l) and get: 

2x(l - x)h" + [(5 -a- {a + 13 + 2){2x - l)]h' = -2n{n + a + [3 + l)h. 

Using the relations (j64p . we obtain: 

2x{l - x)h" + [-5x + 5' {I - x)]h' = -n{2{n - I) + 5 + 6')h. 

Hence, Lh == —anh where L is the infinitesimal generator of ([T]). For n > 0, the sequence 

Qnix) = Pr^'^"\2x - 1) 

is the sequence of eigenfunctions of L. The eigenvalue associated with Qn is — a„. The 
transition operator Pf has the same sequence of eigenfunctions, and the eigenvalues are 
(exp(— Ont)). We have: 

Q„(^) = izi)!!a;-(T-l)(l - a;)-(|-l)-f^[(l - x)"+T-lx"+i-l]. 
ni ax^ 

Each polynomial Qn is of the form (see (jl4p ) 

n 

Qn{x) =^cln-ihLn-i{x). (67) 
i=0 

And each /ijj-j can be developped as 

j 

hij—i = ^ ^ d^"* Qki 

with = Cj^^'^ hij-i{x)Qk{x)TT{x)dx and Cfc = /g^ Q|(x)7r(x)d3;. Since PtQk = 

exp {-akt)Qk, 

j 

Pthij^i = y^exp {-akt)ctk~''Qk- 

k=0 
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This approach requires the computation of the coordinates (it^^ * and of the coefficients c^y^.j 
of ()67p . Our method gives directly the expression of Pthij-i. 

Let us notice that the transition density of ([T]) has the following expression: 

+00 

pt{x, y) = 7r{y) ^ exp {-ant)Qn{x)Qn{y)c~'^ , (68) 

n=0 

as explained in Karlin and Taylor (1981). Therefore, by using the expression (j67p and some 
computations, it is possible to prove that this transition satisfies also condition (Tl) of 
Chaleyat-Maurel and Genon-Catalot (2006). More precisely, this transition can be expressed 
as an infinite mixture of distributions of the class J-. 

This property has the following consequence. Suppose that the initial variable in ([1]) is 
deterministic x(0) = xq. Then, x{ti) has distribution ptj^{xQ,x). This distribution belongs 
to the extended class J- composed of infinite mixtures of distributions of T. We can apply 
our results to the extended class: The filtering, prediction or smoothing distributions all 
belong to ^. 
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